🧠 LLM Inference - emschwartz · Scour

A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints 🏗️LLM Infrastructure

The Inference Shift 🏗️LLM Infrastructure

stratechery.com·1d·Hacker News

Tracing tokens through Llama 3.1 8B inference on H100s 🏗️LLM Infrastructure

krithik.xyz·3d·Hacker News

Understanding KV Cache in LLMs and How It Affects Inference 💾Prompt Caching

pub.towardsai.net

·3d

Reformulating KV Cache Eviction Problem for Long-Context LLM Inference 💾Prompt Caching

Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs 🔬RaBitQ

A General Framework for Multimodal LLM-Based Multimedia Understanding in Large-Scale Recommendation Systems 🎨Chroma

LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models 🏗️LLM Infrastructure

MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference 📏ANN Benchmarks

Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AI 📱Edge AI Optimization

Inference Time Causal Probing in LLMs 🏗️LLM Infrastructure

Understanding Asynchronous Inference Methods for Vision-Language-Action Models 🛡️AI Safety

LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction 💨Cache-Friendly Algorithms

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache 🔬RaBitQ

CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernels 🏗️LLM Infrastructure

Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache 🗂️Vector Indexes

Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation 🏗️LLM Infrastructure

Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs 🏗️LLM Infrastructure

An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference 📦Batch Embeddings

LAWS: Learning from Actual Workloads Symbolically -- A Self-Certifying Parametrized Cache Architecture for Neural Inference, Robotics, and Edge Deployment 📦Batch Embeddings

arxiv.org·5d·Hacker News

Log in to enable infinite scrolling